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System and method for controlling the delay budget of a decoder buffer in a streaming data 
receiver. 



CROSS-REFERENCE TO RELATED APPLICATIONS 

The present invention is related to that disclosed in United States Patent 
Application Serial No. 09/365,463, entitled "DECODER BUFFER FOR STREAMING 
VIDEO RECEIVER AND METHOD OF OPERATION," filed on August 2, 1999, The 
5 foregoing application is commonly assigned to the assignee of the present invention. The 
disclosures of the related patent application is hereby incorporated by reference for all 
purposes as if fully set forth herein. 

TECHNICAL FIELD OF THE INVENTION 
1 0 The present invention is directed, in general, to data streaming systems and, 

more specifically, to a decoder buffer for use in a streaming data receiver, such as a 
streaming video receiver. 

BACKGROUND OF THE INVENTION 

15 Real-time streaming of data, such as multimedia content, over Internet 

protocol (IP) networks has become an increasingly common application in recent years. A 
wide range of interactive and non-interactive multimedia Internet applications, such as news 
on-demand, live TV viewing, video conferencing, and many others, rely on end-to-end 
streaming solutions. Unlike a "downloaded" audio or video file, which may be retrieved first 

20 in "non-real" time and viewed or played back later, streaming audio and streaming video 

applications require an audio source or video source to encode and to transmit an audio signal 
or video signal over a network to an audio receiver or a video receiver, which must decode 
and display (or play back) the transmitted signal in real time. The receiver relies on a decoder 
buffer to receive encoded video data packets and/or encoded audio data packets from the 

25 network and to transfer the packets to a video decoder and/or an audio decoder. 

Two problems arise when a streaming data signal is transmitted across a non- 
guaranteed Quality-of-Service (QoS) network, such as the Internet. First, end-to-end 
variations in the network (e.g., delay jitter) between the streaming data transmitter and the 
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streaming data receiver mean that the end-to-end delay is not constant. Second, there is 
usually a significant packet loss rate across non-QoS networks, often requiring re- 
transmission. The lost data packet must be recovered prior to the time the corresponding 
frame must be decoded. If not, an underflow event occurs. Furthermore, if prediction-based 
5 compression is used, an underflow due to lost data packets may not only impact the current 
frame being processed, but may affect many subsequent frames. 

It is well-known that re-transmission of lost packets is a viable means of 
recovery for continuous media communication over packet networks. Many applications use 
a negative automatic repeat request (NACK) in conjunction with re-transmission of the lost 

1 0 packet. These approaches take into consideration both the round-trip delay and the delay jitter 
between the sender and the receiver(s). 

For example, an end-to-end model with re-transmission for packet voice 
transmission has been developed. This model takes advantage of the fact that voice data 
consists of periods of silence separated by brief talk-spurt segments. The model also assumes 

1 5 that each talk-spurt consists of a fixed number of fixed-size packets. However, this model is 
not general enough to capture the characteristics of compressed video or audio (which can 
have variable number of bytes or packets per video or audio frame). Additionally, an adaptive 
playback algorithm has been developed that changes the playback time of a video frame in 
response to network conditions. This results in a time- varying playback rate (i.e., introduces 

20 "playback jitter") in response to network jitter and packet losses. 

The above mentioned solutions can be applicable for voice data or for certain 
video applications which tolerate "playback jitter." However, these solutions may not be 
acceptable for many types of video-on-demand services (e.g., entertainment applications). In 
addition, while maintaining continuous decoding and displaying of the real-time audio/visual 

25 data, it is crucial for the selected packet loss recovery mechanism to modify its operation 

according to changing conditions during the Internet session in which the data is transmitted. 

Any packet retransmission scheme must strike a balance in determining when 
to request retransmission of a late data packet. If a streaming data receiver waits too long 
before requesting retransmission of a late (and possibly lost) data packet, the requested data 

30 packet may not be received when needed, due to the round trip delay associated with the 

retransmission request and retransmission of the late data packet. However, if the streaming 
data receiver waits only a very brief period before requesting retransmission of a late (but not 
lost) data packet, an excessive amount of the limited bandwidth available between the 
streaming data transmitter and the streaming data receiver will be consumed by the increased 



WO 01/37571 PCT/EPOO/10902 

3 

number of unnecessary retransmission requests and the increased number of duplicate packet 
transmissions. 

There is therefore a need in the art for improved streaming data receivers that 
compensate for variations inherent in a non-QoS network. In particular, there is a need for an 
5 improved receiver decoder buffer that takes into consideration both transport delay 
parameters (e.g., end-to-end delay and delay jitter) and video (or audio) encoder buffer 
constraints. More particularly, there is a need for an improved decoder buffer that 
implements a packet loss recovery mechanism that modifies its operation according to 
changing conditions of the data network over which the streaming data is transmitted and 
1 0 minimizes the number of duplicate data packet transmissions. 

SUMMARY OF THE INVENTION 

The present invention is embodied in an Integrated Transport Decoder (ITD) 
buffer model. One key advantage of the ITD model is that it eliminates the separation of a 

1 5 network-transport buffer, which is typically used for removing delay jitter and recovering lost 
data, from the video/audio decoder buffer. This can significantly reduce the end-to-end delay, 
and optimize the usage of receiver resources (such as memory). 

The present invention provides a re-transmission framework that uses a time- 
delay budget constraint for streaming video receiver during a real-time Internet session. In 

20 other words, at the beginning of the session, the streaming data receiver introduces a certain 
start-up delay to the incoming bitstream. This start-up delay defines the time-delay budget 
that the streaming data receiver can rely on for packet loss recovery for the remainder of the 
session. The re-transmission framework manages this time-delay budget in an adaptive 
manner in response to changing network conditions. The present invention maximizes the 

25 time for uninterrupted decoding and presentation of the multimedia content while minimizing 
time for duplicate-packet transfer events. These duplicate-packet transfer events occur when 
the streaming data receiver requests the re-transmission of packets prematurely, reducing the 
effective available bandwidth between the streaming data transmitter and the streaming data 
receiver. 

30 It is a primary object of the present invention to provide a delay budget 

controller for use with a decoder buffer capable of receiving streaming data packets over a 
data network from a streaming transmitter and storing the data packets in a plurality of access 
units for subsequent retrieval by a streaming data decoder. In an advantageous embodiment, 
the delay budget controller comprises 1) a first controller capable of monitoring at least one 
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network parameter associated with the data network; and 2) a second controller capable of 
monitoring in the decoder buffer a delay budget region comprising a sequence of access units 
that are about to be accessed sequentially by the data decoder, the delay budget region 
comprising a retransmission region and a late region separated by a temporal boundary, 
wherein the second controller detects missing data packets in the retransmission region and 
the late region and, in response to detection of a missing data packet in the retransmission 
region, transmits a retransmission request for the missing data packet to the streaming 
transmitter, and wherein the second controller is capable of adjusting the temporal boundary 
to thereby advance or retard the transmission of the retransmission request. 

In one embodiment of the present invention, the second controller adjusts the 
temporal boundary in response to a measured value of the at least one network parameter. 

In another embodiment of the present invention, the at least one network 
parameter comprises a round trip delay period associated with the retransmission request. 

In still another embodiment of the present invention, the at least one network 
parameter comprises a delay jitter associated with a variation in the round trip delay period. 

In yet another embodiment of the present invention, the at least one network 
parameter comprises an available bandwidth value associated with a communication channel 
between the streaming data transmitter and the decoder buffer. 

In a further embodiment of the present invention, the first and second 
controllers are capable of determining a probability that a packet that is identified as lost by 
the first and second controllers is actually lost. 

In a still further embodiment of the present invention, the second controller 
adjusts the temporal boundary in response to a value of the probability. 

In a yet further embodiment of the present invention, the first controller is 
capable of adjusting a second temporal boundary associated with the delay budget region to 
thereby increase or decrease a duration of the delay budget region. 

Those skilled in the art will readily understand that while the embodiment of 
the present invention described in the DETAILED DESCRIPTION OF THE INVENTION 
that follows is principally oriented towards streaming video (or audio), this is by way of 
illustration only. More broadly speaking, the improved integrated transport decoder buffer 
described below may be readily adapted for use in connection with any type of streaming 
data, including video data and audio data, that must be supplied to a decoder at a required 
rate. 
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The foregoing has outlined rather broadly the features and technical 
advantages of the present invention so that those skilled in the art may better understand the 
detailed description of the invention that follows. Additional features and advantages of the 
invention will be described hereinafter that form the subject of the claims of the invention. 
5 Those skilled in the art should appreciate that they may readily use the conception and the 
specific embodiment disclosed as a basis for modifying or designing other structures for 
carrying out the same purposes of the present invention. Those skilled in the art should also 
realize that such equivalent constructions do not depart from the spirit and scope of the 
invention in its broadest form. 

10 Before undertaking the DETAILED DESCRIPTION, it may be advantageous 

to set forth definitions of certain words and phrases used throughout this patent document: 
the terms "include" and "comprise ," as well as derivatives thereof, mean inclusion without 
limitation; the term "or," is inclusive, meaning and/or; the phrases "associated with" and 
"associated therewith," as well as derivatives thereof, may mean to include, be included 

1 5 within, interconnect with, contain, be contained within, connect to or with, couple to or with, 
be communicable with, cooperate with, interleave, juxtapose, be proximate to, be bound to or 
with, have, have a property of, or the like; and the term "controller" means any device, 
system or part thereof that controls at least one operation, such a device may be implemented 
in hardware, firmware or software, or some combination of at least two of the same. It should 

20 be noted that the functionality associated with any particular controller may be centralized or 
distributed, whether locally or remotely. Definitions for certain words and phrases are 
provided throughout this patent document, those of ordinary skill in the art should understand 
that in many, if not most instances, such definitions apply to prior, as well as future uses of 
such defined words and phrases. 

25 

BRIEF DESCRIPTION OF THE DRAWINGS 

For a more complete understanding of the present invention, and the 
advantages thereof, reference is now made to the following descriptions taken in conjunction 
with the accompanying drawings, wherein like numbers designate like objects, and in which: 
30 FIGURE 1 illustrates an end — to-end transmission of streaming video from a 

streaming video transmitter through a data network to an exemplary streaming video receiver 
according to one embodiment of the present invention; 

FIGURE 2 illustrates an ideal encoder-decoder model of a video coding 

system; 
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FIGURE 3 illustrates end-to-end transmission of streaming video from a 
compressed video source through a channel to an exemplary integrated transport decoder 
buffer and video decoder, according to one embodiment of the present invention; 

FIGURE 4 illustrates a sequence diagram showing the flow of data packets 
through different and distinct regions of exemplary ideal integrated transport decoder buffer; 

FIGURE 5 illustrates an exemplary delay budget controller in accordance with 
one embodiment of the present invention; and 

FIGURE 6 is a flow chart illustrating the operation of an exemplary decoder 
buffer in accordance with one embodiment of the present invention. 

DETAILED DESCRIPTION OR THE INVENTION 

FIGURES 1 through 6, discussed below, and the various embodiments used to 
describe the principles of the present invention in this patent document are by way of 
illustration only and should not be construed in any way to limit the scope of the invention. 
Those skilled in the art will understand that the principles of the present invention may be 
implemented in any suitably arranged streaming video receiver. 

Additionally, those skilled in the art will readily understand that while the 
embodiment of the present invention described below is principally oriented towards 
streaming video, this is by way of illustration only. In fact, the improved integrated transport 
decoder buffer described below may be readily adapted for use in connection with streaming 
audio data or other streaming data that must be supplied to a decoder at a required rate. 

FIGURE 1 illustrates an end — to-end transmission of streaming video from 
streaming video transmitter 110 through data network 120 to at least one streaming video 
receiver 130, according to one embodiment of the present invention. Other streaming video 
receivers similar to streaming video receiver 130 may also be coupled to streaming video 
transmitter 1 10 via data network 120. In order to avoid redundant description, however, these 
other streaming video receivers are not shown or explained. 

Depending on the application, streaming video transmitter 110 may be any one 
of a wide variety of sources of video frames, including a data network server, a television 
station, a cable network, a desktop personal computer (PC), or the like. Streaming video 
transmitter 1 10 comprises video frame source 112, video encoder 1 14 and encoder 
buffer 1 16. Video frame source 1 12 may be any device capable of generating a sequence of 
uncompressed video frames, including a television antenna and receiver unit, a video cassette 
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player, a video camera, a disk storage device capable of storing a "raw" video clip, and the 
like. 

The uncompressed video frames enter video encoder 1 14 at a given picture 
rate (or "streaming rate") and are compressed according to any known compression algorithm 
5 or device, such as an MPEG-4 encoder. Video encoder 1 14 then transmits the compressed 
video frames to encoder buffer 116 for buffering in preparation for transmission across data 
network 120. Data network 120 may be any suitable IP network and may include portions of 
both public data networks, such as the Internet, and private data networks, such as an 
enterprise-owned local area network (LAN) or wide area network (WAN). 
10 Streaming video receiver 130 comprises decoder buffer 131, video 

decoder 134 and video display 136. Decoder buffer 131 receives and stores streaming 
compressed video frames from data network 120. Decoder buffer 131 then transmits the 
compressed video frames to video decoder 134 as required. Video decoder 134 decompresses 
the video frames at the same rate (ideally) at which the video frames were compressed by 
1 5 video encoder 114. 

Decoder buffer 131 further comprises integrated transport decoder (ITD) 
buffer 132, delay budget controller 138 and re-transmission controller 139. In accordance 
with the principles of the present invention, ITD buffer 132 integrates both temporal and 
data-unit occupancy considerations in order to provide video decoder 134 with compressed 
20 video frames at a rate that is sufficient to avoid underflow conditions, during which video 
decoder 134 is starved for compressed video frames. 

ITD buffer 132 accomplishes this in cooperation with delay budget 
controller 138 and re-transmission controller 139. Delay budget controller 138 monitors the 
level of data-occupancy in ITD buffer 132 and detects missing data packets and potential 
25 underflow conditions. In response to notification from delay budget controller 138, re- 
transmission controller 139 requests re-transmission of data missing from ITD buffer 132 in 
order to avoid underflow conditions. 

In an advantageous embodiment of the present invention, ITD buffer 132, 
delay budget controller 138, and re-transmission controller 139 are implemented in a 
30 personal computer (PC) that receives streaming video and/or audio from, for example, the 
Internet over a high-speed data line. In such an embodiment, ITD buffer 132 may be 
implemented in main random access memory (RAM) of the PC or in RAM on a video card, 
and delay budget controller 138 and re-transmission controller 139 may be implemented in 
the CPU of the PC. To implement delay budget controller 138 in a PC environment, the 
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present invention may be embodied as computer executable instructions stored as a program 
on storage media 140. Storage media 140 may be a CD-ROM, a computer diskette, or a 
similar device that may be loaded into removable disk port 141 in streaming video 
receiver 130. 

Continuous decoding of compressed video frames is a key requirement of a 
real-time multimedia application, such as streaming video. To meet this requirement, a 
decoder-encoder buffer model is normally used to ensure that underflow and overflow events 
do not occur. These constraints limit the size (bit- wise) of video pictures that enter the 
encoder buffer. The constraints are usually expressed in terms of encoder-buffer bounds, 
which when adhered to by the encoder, guarantee continuous decoding and presentation of 
the compressed video stream at the receiver. 

FIGURE 2 shows an ideal encoder-decoder model of a video coding system. 
Under this ideal model, uncompressed video frames 201-203 enter the compression engine of 
encoder 214 at a given picture-rate, X frames/second, as indicated by the Time(l) line. The 
compressed frames exit encoder 214 and enter encoder buffer 216 at the same X 
frames/second, as indicated by the Time(2) line. Similarly, the compressed frames exit 
decoder buffer 216 and enter channel 220 at X frames/second. Channel 220 is a generic 
representation of any transmission medium, such as the Internet, that transfers compressed 
video frames from a transmitting source to a receiver. In the ideal case, the delay of 
channel 220 (5c) is a constant value. 

Next, the compressed frames exit channel 220 and enter decoder buffer 232 at 
the same X frames/second as at the input and the output of encoder 214, as indicated by the 
Time(3) line. Decoder buffer 232 transmits the compressed frames to decoder 234, which 
decompresses the frames and outputs decompressed frames 251-253 at the original X 
frames/second at which frames entered encoder 214. 

Ideally, the end-to-end buffering delay (i.e., the total delay encountered in both 
encoder buffer 216 and decoder buffer 232) is constant. However, the same piece of 
compressed video data (e.g., a particular byte of the video stream) encounters different delays 
in encoder buffer 216 and decoder buffer 232. In the ideal model, encoding in encoder 214 
and decoding in decoder 234 are instantaneous and require zero execution time and data 
packets are not lost. 

The encoder buffer bounds can be expressed using discrete-time summation. 
In discrete-time domain analysis, A is the end-to-end delay (i.e., including both encoder 
buffer 216 and decoder buffer 232 and channel 8 C ) in units of time. For a given video coding 
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system, 5 is a constant number applicable to all frames entering the encoder-decoder buffer 
model. 

To simplify the discrete-time analysis, it is assumed that the end-to-end 
buffering delay (AT = A-5 C ) is an integer-multiple of the frame duration (T). Therefore, NA = 
N (A-5 C )/T represents the delay of the encoder and decoder buffers in terms of the number of 
video frames (N). For the purposes of clarity and brevity in describing the principles of the 
present invention, the remainder of this disclosure will use time units specified in frame- 
duration intervals. For example, using the encoder time reference shown in FIGURE 2, the 
n* frame enters encoder buffer 216 at time index "n". The decoder time-reference of decoder 
buffer 232 is shifted by the channel delay (5 C) . with respect to encoder buffer 216. 

The data rate (r) at the output of encoder (e) 214 during frame-interval "i" may 
be represented as r e (i). Here, "data rate" is used generically. It could signify bit rate, byte rate, 
or even packet rate. Similarly, the data rate at the input of decoder buffer 232 may be 
represented as r d (i). Based on the ideal model, r e (iT) = r d (iT+D c ). In addition, based on the 
convention established above, r e (i) = r d (i). Thus, the bounds of encoder buffer 216 can be 
expressed as: 



max 



n + AW 



max 



,o 





( n+tiN ^ 




< B e (n) < min 













[EQUATION 1] 



where B* ax and s* ax are the maximum decoder and encoder buffer sizes respectively. 

In the ideal case, it is assumed that encoder 214 begins to transmit data 
immediately after the first frame enters encoder 214. Therefore, the start-up delay dd f (i.e., 
the delay time the first piece of data from the first picture spends in decoder buffer 232 prior 
to decoding) equals the end-to-end, encoder-decoder buffer delay: dd f = AT = TAN. 

In one embodiment of the present invention, ITD buffer 132 minimizes 
underflow events by taking into consideration the above-described problems of the ideal 
buffer model and the ideal encoder-decoder buffer constraints. ITD buffer 132 is based on 
lost packet recovery using re-transmission. 

FIGURE 3 is a simplified block diagram of exemplary end-to-end 
transmission of streaming video, without support for re-transmission. For the purposes of 
simplicity and clarity, streaming video transmitter 1 1 0 has been replaced by compressed 
video source 305 and data network 120 has been replaced by channel 320. Compressed video 
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source 305 transmits data packets at rate r e (n) and channel 320 transmits data packets at rate 
r td (n). Since video re-transmission is not discussed in FIGURE 3, delay budget controller 138 
and re-transmission controller 139 are omitted from the diagram. Streaming video receiver 
130 has been simplified and is represented by ITD buffer 132 and video decoder 134. 

As noted above, ITD buffer 132 integrates temporal and data-unit occupancy 
models. ITD buffer 132 is divided into temporal segments of T' seconds each. By way of 
example, the parameter T may be the frame period in a video sequence. The data packets 
(bits, bytes, or packets) associated with a given duration T are buffered in the corresponding 
temporal segment. All of the data packets associated with a temporal unit are referred to as an 
"access" unit. By way of example, data packets 351, 352, and 353 comprise access unit A n +i, 
data packet 354 comprises access unit A n+2 , and data packets 355 and 356 comprise access 
unit A n+3 . 

During time interval n, the access unit, A n , is being decoded by 
decoder 1 34 and access unit A n +i is stored at the temporal segment nearest to the output of 
ITD buffer 132. An access unit may be an audio frame, a video frame, or even a portion of a 
video frame, such as Group of Blocks (GOB). Therefore, the duration required to decode or 
to display an access unit is the same as the duration of the temporal segment T. During the 
time-interval n, the rate at which data enters ITD buffer 132 is r td (n). The numbers of data 
packets in each access unit are not required to be the same. Compression algorithms used in 
video encoder 1 14 may compress the data packets in successive access units by different 
amounts, even though each access unit represents temporal units of the same duration. 

For example, the three data packets 351-353 in access unit A n +i may comprise 
a complete video frame, Frame 1 . The single data packet 354 in A n +2 may represent only 
those portions of Frame 2 that are different than Frame 1. Nonetheless, data packet 354 is 
sufficient to create Frame 2 if the Frame 1 data is already known. Since Frame 1 and Frame 2 
have the same duration, the temporal segment, T, is the same for A n +i and A n +2- 

Each temporal segment holds a maximum number of packets, K max , with each 
packet having a maximum size, b max , (in bits or bytes). Therefore, the maximum size of an 
access unit, S max , may be represented by S max ^ K^Ow). Video encoder 1 14 is assumed to 
begin each access-unit with a new packet that is present only in that access unit. 

The amount of data in ITD buffer 132 at time index n, B td (n), may be 
described by terms of B a (n) and B b (n). B a (n) represents the number of consecutive-and- 
complete access units in ITD buffer 132 at the beginning of interval n, and B b (n) represents 
the total consecutive amount of data in ITD buffer 132 at the end of interval n. For B a (n), 
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temporal segments containing partial data are not counted, and all segments following a 
partial segment are also not counted even if they contain a complete, access-unit worth of 
data. Hence, TB a (n) represents the amount of video data in temporal units (e.g., seconds) that 
ITD buffer 132 holds at time index n (without running into an underflow if no more data 
5 arrives). 

If S n denotes the size of access unit n, the relationship between B a and B b can 
be expressed as Equation 2 below: 

J=N+1 [EQUATION 2] 

where S, is the maximum size of the access unit for temporal segment j and U B a („)+] is the 
[ 0 partial (incomplete) data of access unit A n+ B a (n)+1 stored in temporal segment B a (n)+1 at the 
beginning of time index n. 

In order for re-transmission to occur, ITD buffer 132 must be able: a) to output 
one temporal segment (T) worth of data at the beginning of every temporal time-interval n; 
b) to detect lost packet(s) and to transmit corresponding negative acknowledge (NACK) 
5 messages to transmitter 1 1 0 or compressed video source 305; c) to continuously store newly 
arrived primary (i.e., not re-transmitted) packets; and d) to store re-transmitted packets. Ideal 
ITD buffer 132 maintains the data rate of the video stream, without delays caused by re- 
transmission of any lost data. In other words, if r e (n) is the transmission data rate used by 
idealized video encoder 1 14 under lossless circumstances, ideal ITD buffer 132 will maintain 
this date rate without degradation caused by the re-transmission process. Depending upon the 
number of re-transmission requests, encoder buffer 116 may adjust its output data rate r e (n), 
with a corresponding adjustment by ITD buffer 132. 

In one embodiment, decoder buffer 131 adds buffering for the incoming video 
stream in order to compensate for the time required for detection and recovery of lost data 
and for the delay associated with a "real" world implementation. By delaying all incoming 
video streams by this compensation time, decoder buffer 131 outputs video stream data at a 
continuous rate as required for decoding. The minimum duration of time needed for detecting 
a predetermined number of lost packets is represented by T L . In general, T L is a function of 
the delay jitter caused by data arriving later than expected by ITD buffer 132. 

The minimum amount of time needed for streaming video receiver 1 30 to 
recover a packet after being declared lost is represented by T R . Time T R includes the time 
required for streaming video receiver 130 to send a NACK to streaming video transmitter 1 10 
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and the time needed for the re-transmitted data to reach streaming video receiver 130 
(assuming that the NACK and re-transmitted data are not lost). 

Exemplary decoder buffer 131 transfers a re-transmitted packet with a 
minimum delay (T L +T R ) for the lost packet interval. If the minimum delay experienced by 
5 any video data for an ideal decoder buffer 1 3 1 is represented by dd min , the amount of delay \ 
that may be added to the minimum ideal delay in order to account for the total delay for re- 
transmission is: 

A* > u(T L +t r - dd min ) [EQUATION 3] 
where u(x)=x for x>0, and u(x)=0 for x < 0. 
1 0 Decoder buffer 1 3 1 adds delay A R buffering for all output data to video 

decoder 134 in order to provide time for decoding and transferring of the data, resulting in 
continuous video streams. Therefore, the total encoder buffer 1 16 to decoder buffer 132 
output delay (A TO t) may be represented by: 

Atot = Ajdeai + A R > A ide ai + u(T L +T R -dd min ) [EQUATION 4]. 
1 5 ITD buffer 1 32 provides buffering (storage) for a minimum number of 

temporal segments ( B* in ) as compensation for re-transmission time requirements and as 

prevention for an underflow event. The ITD buffer 132 sizing may be based, for example, on 
minimum and maximum boundaries for storing temporal segments. The process for 
determining these boundaries was described in United States Patent Application Serial No. 
20 09/365,463, entitled "DECODER BUFFER FOR STREAMING VIDEO RECEIVER AND 
METHOD OF OPERATION" and previously incorporated by reference into this disclosure. 

FIGURE 4 is a sequence diagram showing the flow of data packets through 
different regions of exemplary ITD buffer 132 under the assumption that dd min = 0 (the lower 
boundary level) and d max = A ide ai. ITD buffer 132 data enters from the right side of the 
25 diagram and exits to the video decoder 1 34 at the left side. The most recently received data is 
in a buffer area which is labeled "too-early for re-transmission request region" (too-early). 
Depending on the location in the too-early region of the buffer, ITD buffer 132 introduces 
buffer delays labeled N E , AN, or N L . The area of this too-early buffer region, which comprises 
the ideal delay AN, is labeled as the ideal-buffer region. ITD buffer 132 manages the ideal- 
30 buffer region as an ideal video buffer. In other words, data packets flow through this region 
and are only delayed by the inherent characteristics of the buffer elements). Ideal ITD buffer 
132 provides the remaining too-early buffer areas to compensate for delays associated with 
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the transfer of video streams from streaming video transmitter 1 10 to decoder 131 (N E ), as 
well as delays caused by delayed or lost video packets (Nl). 

ITD buffer 132 provides a delay, N R , in the re-transmission region in order to 
compensate for expected time requirements for the initiation and reception of re-transmission 
requests. Exemplary decoder buffer 131 initiates re-transmission requests during the time 
periods associated with the re-transmission region. 

It is important to note that the ideal-buffer and re-transmission regions may 
overlap, depending on the values of the different delay parameters (dd min , T R , T L ). However, 
for the exemplary ideal ITD buffer 132 with dd min =0, the re-transmission and ideal-buffer 
regions do not overlap. 

For ITD buffer 132, N E represents the initial decoding delay (dd f ) which 
corresponds to the amount of delay encountered by the very first piece of data that enters the 
buffer prior to the decoding of the first picture (or access unit). This dd f is based on, among 
other things, the streaming video transmitter 1 10 and data network 120 data transmission 
rates during elapsed time dd f . In the ideal case, ITD buffer 132 uses this same data rate for 
entering received data into its buffer (storage) regions. Ideal decoder buffer 131 recognizes 
the amount of data in its ITD buffer 132 regions just prior to the time that the first access unit 
is decoded as B d 0 data. The B d 0 data, also referred to as "start-up-delay" data, is given by the 
following relationship: 

AN 

^ [EQUATION 5]. 

When dd m j n =0 5 ideal decoder buffer 131 re-transmission processing is 
comprised of the following procedures: 

1 . The ideal-buffer region is filled until all data associated with the start-up 
delay are in the buffer. Since lost events may also occur during this time interval, these data 
may be treated in a special way, such as by using reliable transmission (e.g., using TCP) for 
them. The ideal condition for lossless data is satisfied when: 

*= Wjt+ *» t+ i [EQUATION 6] 

where B k is the amount of data stored in ideal ITD buffer 132 temporal segment k at any 
instant of time. 

2. After Equation 15 is satisfied, ITD buffer 132 advances the content of all 
temporal storage segments by one segment toward the buffer output. Subsequently, ideal ITD 
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buffer 132 repeats this process every T units of time. After N L +N R periods of T (i.e., after 
Tl+Tr), decoder 134 starts decoding the first access unit. The time-period that starts when 
decoding of the first access unit begins is labeled TV Hence, the beginning of any time period 
n (T n ) represents the time when access unit A n +k is moved to temporal segment k. 
5 3. Ideal ITD buffer 132 considers data missing in temporal segment Nr of the 

re-transmission buffer region as lost. This condition occurs when: 

B„ a W < S n+Nfi [E q UAXIO N7] 
where Bn(r> is the amount of data in temporal segment Nr at time period n and Sj is the size 
of access unit j. When ideal ITD buffer 132 determines that data is missing, it sends a re- 

1 0 transmission request to streaming video transmitter 110. 

4. Ideal ITD buffer 132 places arriving re-transmitted data into their 
corresponding temporal segments of the re-transmission region. Assuming the re-transmitted 
data are received, ideal ITD buffer 132 transfers the re-transmitted data to the video decoder 
134 prior to the decoding times of their corresponding access units. 

15 In order to properly balance the need to request retransmission of late packets 

in a timely manner with the need to avoid duplicative data packet deliveries, the present 
invention provides a retransmission framework that uses a time-delay budget constraint for 
streaming data receivers during real-time Internet sessions. Delay budget controller 138 
manages this time-delay budget in an adaptive manner in response to changing network 

20 conditions, as described below in greater detail. 

The present invention is primarily receiver-oriented retransmission technique. 
FIGURE 5 illustrates exemplary delay budget controller 138 in accordance with one 
embodiment of the present invention. Delay budget controller 138 comprises real-time QoS 
characterization circuit 505, buffer management circuit 510, and delay budget management 

25 circuit 515, which coordinate the flow of streaming video between streaming video receiver 
130 and streaming video transmitter 110. 

In one embodiment of the present, delay budget controller 138 establishes a 
delay budget that is static and adapts the sizes of different temporal regions within the static 
delay budget. In an alternate embodiment of the present invention, delay budget controller 

30 138 may establish a delay budget that is dynamic and may adapt the size of the over all delay 
budget, as well as adapt the size of the different temporal regions within the dynamic delay 
budget. 
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This is illustrated in FIGURE 4. The temporal position of one or both of 
boundary lines 405 and 410 may be adjusted by delay budget controller 138, as indicated by 
the double arrow lines 406 and 41 1 . In a static mode, delay budget controller 138 does not 
adjust the position of boundary line 410, so that the delay budget = N R + N L is constant. For 
5 example, if Nr is initially selected so that buffer region corresponding to N R is 2 seconds and 
N L is selected so that the buffer region corresponding to Nl is 0.5 seconds, then the delay 
budget is held constant at 2.5 seconds by delay controller 138. Delay controller 138 may still 
adjust the position of boundary line 405 within the delay budget, but the total value of the 
delay budget remains fixed at 2.5 seconds. In a dynamic mode, delay budget controller 138 

1 0 may adjust the position of boundary line 41 0, so that the delay budget comprises N R , N L , and 
at least part of AN. Delay budget controller 138 may also adjust the position of boundary line 
405 in the dynamic mode. 

Real-time QoS characterization circuit 505 continuously calculates and 
updates the characterization of Internet sessions in real-time using three network parameters: 

1 5 round-trip delay (S^t)), delay jitter (5 jit (t)), and bandwidth (P(t)). An exemplary network- 
based random process model K may then be derived based on a set of observations, 0(t) = 
{8 jit (t), 5rtt(t), P(t)}, using these three QoS parameters. Once calculated, real-time QoS 
characterization circuit 505 communicates the network QoS model set for a particular time t, 
K(t;0), to other real-time processes in streaming video receiver 130: 

20 K(t;0) = {5j it (t), 8 rtt (t), P(t)} [EQUATION 8] 

Although real-time QoS characterization circuit 505 primarily determines QoS 
characterization parameters, assistance with these calculations may occasionally be needed 
from streaming video transmitter 110. 

Buffer management circuit 510 identifies internal and external delay times and 

25 manages the flow of data for ITD buffer 132. At the beginning of each streaming session, 

buffer management circuit 510 identifies two minimum delay parameters for buffering media 
streams. The delay parameters are the minimum ideal start-up delay, A su > and the minimum 
total start-up delay, D su . The value A su represents the minimum buffering delay needed to 
ensure that a multimedia video clip will not encounter an underflow condition during the 

30 streaming session (assuming no network delay jitter). Since A su is a function of the particular 
media stream, its value can be communicated to buffer management circuit 510 by different 
means. For instance, A su information may be carried in the media stream, as in the case with 
the MPEG-2 transport stream and PES layer time stamps. 
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Buffer management circuit 510 may identify or compute the value for D su in 
several ways. One criterion is that D su should always be larger than A su . In one embodiment, 
buffer management circuit 510 may obtain the value for D su from initialization data which 
may be modified by the user. In another embodiment, buffer management circuit 510 may 
5 access a programmable or user entered maximum start-up delay which can be substituted for 
D su or used as the basis for calculating D su . 

In another embodiment, buffer management circuit 510 may calculate 
preliminary estimates for the round-trip delay 5 m (t), delay jitter 5j it (t), and the bandwidth p(t), 
while ITD buffer 132 is storing streaming data during the initial ideal A su period. Buffer 
10 management circuit 510 may then use these QoS parameters for calculating a desirable value 
for D su based on some criterion (e.g., pick a minimum D su value that enables decoder buffer 
131 to recover lost packets with high probability of success). An exemplary buffer 
management circuit 510 computation for D su based on preliminary QoS characterization 
estimates is described below. Alternatively, buffer management circuit 510 may use a 
1 5 combination of user/application driven or network-condition driven applications, or other 
approaches, for computing D su . 

After A su and D su have been identified, budget management circuit 510 
calculates the minimum delay budget (D bu dget)> where Dbudgct = Dsu - A su - The Dbudget 
represents the minimum amount of time available for detecting and recovering lost packets 
20 for the remainder of the data streaming session. 

Once initialized and after receiving required time allocations for detecting and 
recovering lost packets, buffer management circuit 510 monitors ITD buffer 132 for the 
absence of expected packets. A minimum monitoring detection time (x de t) determines the 
time before a NACK is sent to streaming video transmitter 110. Buffer management circuit 
25 510 may transfer from 1 to R NACKS for each temporal segment of "T" seconds, where R 
represents the number of access units present in T. 

Delay-budget management circuit 515 determines and manages the elapsed 
time needed for detecting and for recovering lost packets through the re-transmission process. 
In one embodiment, delay-budget management circuit 515 accomplishes these 
30 responsibilities by allocating portions of the time calculated for a static D bu d g et as packet loss 
detection time (tact) and as packet re-transmission time (xnx) so that Dbudget ~ Xd et + x^. 

Delay-budget management circuit 515 takes into consideration the real-time 
network condition model K derived by real-time QoS characterization circuit 505: 
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Tdet (t; K) + (t; X) - D budget [EQUATION 9] 
The above equation expresses the constraints that delay-budget management 515 circuit must 
adhere to when using a static value for D bu dget. A generalization of this static approach to a 
time-varying (or dynamic) delay-budget framework is described below. 

There are a number of ways of determining T det and Tnx. In one embodiment of 
the present invention, during a session delay-budget management circuit 5 1 5 continuously 
adapts to changing network conditions by generating the current estimates of positive delay 
jitter Ajit+(t) and the round-trip delay A m (t) using the set of observations 0(t) = {5ji t (t),8 m (t)} 
that were collected in real time. Once estimates of Aj it +(t) and A m (t) are determined, delay- 
budget management circuit 515 may scale these values to fit into the delay budget and 
assigns the scaled values to T det (t) and ^(t) as follows: 

Trtx(t) = [w m (t)A rtt (t)/(w rtt (t)A r tt(t)+W jit+ (t)Aj it+ (t))]D budg et 

and 

Tdct(t) = [w jil+ (t)A jit+ (ty^^ 

For a given pair of estimates, A,tt(t) and Aj it +(t), if the weighting factors w m (t) 
and Wju+(t) are equal, these formulas give equal weight to both the packet-loss detection 
process and the retransmission process. Therefore, the delay budget, D budge t, is utilized in a 
"fair" manner between the two processes. This is accomplished by keeping the ratio 
Xrtx(t)/T de t(t) the same as A m (t)/ A jit +(t) for all times t. 

Delay-budget management circuit 515 uses a packet-lost time allocation 
process that minimizes the time needed for declaring a packet lost, while providing enough 
time so that the probability that the packet is really lost is as high as possible. The latter 
criterion is important for minimizing the duplicate transfer of packets and increasing the 
effective available bandwidth over the Internet session under consideration. 

Once the packet-lost time (T del ) has been allocated, delay-budget management 
circuit 515 allocates the remaining time of Db udg et as time for recovering the lost packet. Thus, 
the time allocated for detecting a lost packet is based on minimizing the time for detecting a 
lost packet, while maximizing the available time for recovering the lost packet. When 
successful, the lost packet is recovered prior to the time when it is needed by video decoder 
134. 

In some cases, streaming video receiver 130 is constrained to operate within a 
maximum start-up delay due, for example, to user/application requirements. In these cases, 
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buffer management circuit 510 treats the determination of D su as trivial and only considers 
the minimum (ideal) start-up delay A su in its calculations. 

When buffer management circuit 510 is not constrained by a strict maximum 
start-up delay, it may limit the time for the start-up delay in order to improve, for example, 
the system response time to the user or to use system resources (e.g., memory) more 
efficiently. In one embodiment, buffer management circuit 510 estimates start-up delay using 
a probabilistic approach based on generating preliminary models for the QoS parameters. 
These preliminary models may be generated by buffer management circuit 510 during the 
initial (ideal) A su , with consideration given to prior knowledge or assumptions about the 
characteristics of the Internet session under consideration. In addition, buffer management 
circuit 510 bases its estimate for D su on a desirable success ratio for packet loss recovery and 
a desirable confidence level in accurately identifying packet-loss occurrences. 

For this embodiment, P R represents the desirable probability for successfully 
recovering a lost packet (i.e., recovering the packet prior to the time it is needed by the 
decoder). Similarly, P L represents the probability that a packet has been lost. Based on the 
ITD buffer model introduced in United States Patent Application Serial No. 09/365,463, 
buffer management 510 computes D su as follows: 

D su = (P R ) + P;t (P L ) + A su [EQUATION 1 0] 

where P„, (6) and P jU . (S) are the probability distribution functions of the round-trip delay 

and the positive delay-jitter parameters, respectively. P~ } (•) and PJ^ (-) are the 

corresponding inverse functions for P m (S) and P (S) . 

Using these probability functions, buffer management circuit 510 measures 
positive-jitter events only for packets arriving later than expected. Consequently, buffer 
management circuit 5 1 0 derives the positive-jitter density-function p (5) by normalizing 
the overall jitter probability-density function: 

Pjr (8) = p. u (S)/[l - P JU (0)] [EQUATION 11] 

For these calculations, buffer management circuit 510 assumes that 5 m represents the total 
time for recovering a lost packet or the time interval from when the packet is declared lost 
through the re-transmission time. Moreover* buffer management circuit 510 assumes that the 
positive delay-jitter as received from data network 120 represents the time needed for 
declaring a packet lost (6ji t (t)). . 
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As mentioned previously, delay-budget management circuit 515 ensures a 
high confidence level for determining the amount of time allocated for identification of lost 
packets in order to minimize the transfer of duplicate packets. This high confidence level is 
especially needed when the available bandwidth is virtually 100% occupied by the 
multimedia stream being transmitted. 

However, in cases when the available bandwidth p(t) is higher than the 
bitstream bit rate r(t), delay-budget management circuit 515 may utilize some of the excess 
bandwidth [i(t)-r(t)] for increasing the probability of recovering lost packets. In these cases, 
delay-budget management circuit 5 1 5 may decrease the value for 5ji t (t) which may cause an 
increase of duplicate packet transmissions. When excess bandwidth is present, these 
duplicated packets will not compete with the primary bitstream. The extra time available for 
re-transmission of lost-packets may improve the overall packet-loss-recovery mechanism 
without causing a significant increase in the packet loss ratio. 

In one embodiment, delay-budget management circuit 515 enhances the ability 
to calculate a minimum value for 5ji t across time t by considering the change in excess 
bandwidth across time and the desirable probability for successful recovery of a lost packet. 

Delay-budget management circuit 5 1 5 determines that a packet is lost when 
the following equation is satisfied: 

P l = j/V dS [EQUATION 12] 
o 

where P L , in this case, represents the probability that a packet has been lost when it is 
declared lost. 

In other words, when the probability of a packet-loss event for a particular 
packet reaches P L , delay-budget management circuit 515 declares the packet lost by the 
system. In a static implementation for P L , delay-budget management circuit 515 simply sets 
Pl to equal the desired static confidence level. 

Delay-budget management circuit 515 modifies the value of P L in response to 
the network condition and to the characteristic of the stream bit rate. In one embodiment, 
delay-budget management circuit 515 quantifies the change of P L across time (PlO) as 
follows: 

P l(0= \PjA S ^ dS * [EQUATION 13] 

6 P(t) 
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Using this relationship, delay-budget management circuit 515 chooses a 
smaller P L as the trigger for a packet-loss declaration event when the available bandwidth is 
high. 

As explained above, in the absence of any positive delay-jitter in the network, 
the delay-budget management circuit 515 may rely on the minimum delay D budget for packet 
loss recovery. This is due to the fact that the ideal decoding delay of the media stream can be 
as small as zero during any given session. For example, the Integrated Transport Decoder 
(ITD) buffer model described in United States Patent Application Serial No. 09/365,463 
incorporates an ideal decoder section that can be drained even in the absence of network jitter 
or packet loss events. However, this ideal decoder section can include a delay Adec(t) that 
varies in time and can be as large as the end-to-end encoder-decoder ideal buffering delay A. 

For the static embodiment of delay-budget management circuit 515, delay- 
budget management circuit 515 only focuses on and manages the re-transmission section of 
ITD buffer 132. This embodiment provides the means for simplifying ITD buffer 132 and the 
time-delay management process. 

In another embodiment which incorporates a relatively small increase in 
complexity, streaming video receiver 130 uses a dynamic time-varying delay budget for its 
packet loss recovery: 

D(t) = D bud get + A d ec(t) [EQUATION 14] 
where D bu d g et is the same parameter as briefly described for the static embodiment of delay- 
budget management 515. One dynamic embodiment of delay-budget management circuit 515 
continuously monitors the ideal-buffer section in order to estimate Ad ec (t). Furthermore, the 
delay-budget constraint expressed in EQUATION 8 is modified as follows: 

Tdet (t; X) + (t; K) = D(t) [EQUATION 1 5] 

where: 

Dbudget < D(t) < Dbudget + A [EQUATION 16] 
This dynamic framework can be further generalized to include multiple 
attempts for packet loss recovery. In this case, depending on the amount of delay available, 
buffer management circuit 510 can request n packet re-transmission times: 

n(t) x det (t; K) + (t; K)] = D(t) [EQUATION 10] 
Therefore, as the occupancy of ITD buffer 132 changes, the buffer management circuit 510 
can change n. This capability approaches the benefits achieved from the instantaneous delay 
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Adec(t) experienced in the ideal-buffer section of the ITD buffer model introduced in United 
States Patent Application Serial No. 09/365,463. 

FIGURE 6 is a flow chart illustrating the operation of exemplary decoder 
buffer 13 1 in accordance with one embodiment of the present invention. Initially, delay 
budget management circuit 5 1 5 identifies a minimum total start-up delay (D S u) and a 
minimum ideal start-up delay (A S u), and calculates a delay budget (Dbudget) (process step 
605). Real-time QoS characterization circuit 505 monitors conditions on data network 120 
and collects measured data for round-trip delay, 5 m (t), delay jitter, 8j it (t), and bandwidth, p(t) 
(process step 610). 

Delay budget controller 138 monitors ITD buffer 132 to detect the absence of 
expected packets. If a packet is not received after a minimum monitoring time, it is 
determined to be lost and a NACK message is sent to streaming video transmitter 110 
(process step 615). Using the statistics gathered by real-time QoS characterization circuit 
505, delay budget controller 138 calculates the probability distribution functions for round 
trip delay and delay jitter. Delay budget controller 138 also calculates the probability that a 
video packet has actually been lost when it was declared lost (process step 620). 

If the probability that a particular data packet has been lost exceeds an 
established probability threshold, delay budget controller 138 transmits a retransmission 
request to streaming video transmitter 1 10 (process step 625). Optionally, delay budget 
controller 138 may adjust the value of the established probability threshold in order to adapt 
to changing delay conditions on network 120 (process step 630). 

Although the present invention has been described in detail, those skilled in 
the art should understand that they can make various changes, substitutions and alterations 
herein without departing from the spirit and scope of the invention in its broadest form. 
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1 • For use with a decoder buffer (131) capable of receiving streaming data 

packets over a data network (120) from a streaming transmitter (1 10) and storing said data 
packets in a plurality of access units for subsequent retrieval by a streaming data decoder 
(134), a delay budget controller (138) comprising: 

a first controller (505) capable of monitoring at least one network parameter 
associated with said data network (120); 

a second controller (515) capable of monitoring in said decoder buffer (131) a 
delay budget region comprising a sequence of access units that are about to be accessed 
sequentially by said data decoder (134), said delay budget region comprising a retransmission 
region and a late region separated by a temporal boundary, wherein said second controller 
(515) detects missing data packets in said retransmission region and said late region and, in 
response to detection of a missing data packet in said retransmission region, transmits a 
retransmission request for said missing data packet to said streaming transmitter (110), and 
wherein said second controller (515) is capable of adjusting said temporal boundary to 
thereby advance or retard said transmission of said retransmission request. 

2 - The delay budget controller (138) set forth in Claim 1 wherein said second 
controller (515) adjusts said temporal boundary in response to a measured value of said at 
least one network parameter. 

3 - The delay budget controller (138) set forth in Claim 2 wherein said at least one 
network parameter comprises a round trip delay period associated with said retransmission 
request. 

4 - The delay budget controller (138) set forth in Claim 3 wherein said at least one 
network parameter comprises a delay jitter associated with a variation in transmission delay 
from said streaming transmitter ( 1 1 0) to said decoder buffer (131). 
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5- The delay budget controller (138) set forth in Claim 2 wherein said at least one 

network parameter comprises an available bandwidth value associated with a communication 
channel (120) between said streaming data transmitter and said decoder buffer (131). 

6. The delay budget controller (138) set forth in Claim 2 wherein said first and 

second controllers (505, 515) are capable of determining a probability that a packet that is 
identified as lost by said first and second controllers (515) is actually lost. 

7- The delay budget controller (138) set forth in Claim 6 wherein said second 

controller (515) adjusts said temporal boundary in response to a value of said probability. 

8. The delay budget controller (138) set forth in Claim 2 wherein said second 
controller (515) is capable of adjusting a second temporal boundary associated with said 
delay budget region to thereby increase or decrease a duration of said delay budget region. 

9. A video system comprising: 

a decoder buffer (131) capable of receiving streaming video packets over a 
data network (120) from a streaming video transmitter and storing said video packets in a 
plurality of access units for subsequent retrieval; 

a delay budget controller (138) comprising: 

a first controller (505) capable of monitoring at least one network parameter 
associated with said data network (120); 

a second controller (515) capable of monitoring in said decoder buffer (13 1) a 
delay budget region comprising a sequence of access units that are about to be accessed 
sequentially, said delay budget region comprising a retransmission region and a late region 
separated by a temporal boundary, wherein said second controller (515) detects missing 
video packets in said retransmission region and said late region and, in response to detection 
of a missing video packet in said retransmission region, transmits a retransmission request for 
said missing video packet to said streaming transmitter (110), and wherein said second 
controller (515) is capable of adjusting said temporal boundary to thereby advance or retard 
said transmission of said retransmission request; 

a video decoder capable of retrieving said video packets from said decoder 
buffer (131) and decoding said video packets; and 
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a display device coupled to said video decoder for displaying said decoded 

video packets. 

1 °- For use with a decoder buffer (131) capable of receiving streaming data 

packets over a data network (120) from a streaming transmitter (110) and storing the data 
packets in a plurality of access units for subsequent retrieval by a streaming data decoder 
(134), a method of controlling retransmission requests associated with lost ones of the data 
packets comprising the steps of: 

monitoring at least one network parameter associated with the data network 

(120); 

monitoring in the decoder buffer (131) a delay budget region comprising a 
sequence of access units that are about to be accessed sequentially by the data decoder (134), 
the delay budget region comprising a retransmission region and a late region separated by a 
temporal boundary; 

detecting missing data packets in the retransmission region and the late region; 

in response to detection of a missing data packet in the retransmission region, 
transmitting a retransmission request for the missing data packet to the streaming transmitter 
(110); and 

adjusting the temporal boundary to thereby advance or retard the transmission 
of the retransmission request. 

1 1 • For use with a decoder buffer (131) capable of receiving streaming data 

packets over a data network (120) from a streaming transmitter (1 10) and storing the data 
packets in a plurality of access units for subsequent retrieval by a streaming data decoder 
(134), computer-executable process steps stored on a computer-readable storage medium 
(140) for controlling retransmission requests associated with lost ones of the data packets 
comprising the steps of: 

monitoring at least one network parameter associated with the data network 

(120); 

monitoring in the decoder buffer (131) a delay budget region comprising a 
sequence of access units that are about to be accessed sequentially by the data decoder (134), 
the delay budget region comprising a retransmission region and a late region separated by a 
temporal boundary; 

detecting missing data packets in the retransmission region and the late region; 
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in response to detection of a missing data packet in the retransmission region, 
transmitting a retransmission request for the missing data packet to the streaming transmitter 
(110); and 

adjusting the temporal boundary to thereby advance or retard the transmission 
of the retransmission request. 
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